In a computer system, there are various types of “interrupts,” which may be considered as requests for attention for a processor. Typically, when the processor receives an interrupt, it suspends its current operations, saves the status of its work, and transfers control to a special routine. An interrupt handler may be utilized to process instructions for a particular interrupt. Interrupts can be generated, for example, by various hardware devices to request service or report problems, or by the processor itself in response to program errors or requests for operating-system services.
Input/output (I/O) interrupts are typically generated on the basis of requesting attention to an I/O completion. Even if consideration is restricted to I/O interrupts, there are different types. Physical I/O interrupts are delivered in both a physical system and a virtualized system. A virtualized computer system will be described briefly below (when referring to FIG. 1), but a range of such systems exist. For example, various virtualized systems are commercially available from VMware, Inc., the assignee of this patent document. Virtual I/O interrupts occur routinely within a virtualized or para-virtualized system.
In addition, an I/O completion may trigger an inter-processor interrupt (IPI). IPIs allow one processor to interrupt a second processor within a multiprocessor system. An IPI may be utilized in either a virtualized or non-virtualized system when an interrupt is issued to the first processor for an event that is relevant to the second processor. For example, if an I/O interrupt is directed to a first processor, but the first processor is not running the targeted Virtual Machine (VM) having interest in the I/O completion, the interrupt is redirected. The interrupt is steered to the second processor which is running the targeted VM. This software-based interrupt steering often causes a reduction in performance. Thus, IPIs are “expensive.”
FIG. 1 illustrates one generalized virtualized computer system. The illustrated elements may be substantially the same as corresponding elements of other VMware patents and applications. The virtualized computer system of FIG. 1 comprises system hardware 10, which interfaces with one or more disks 12 (or other storage media). System hardware 10 comprises memory 14 and a storage adapter 16 appropriate for the disk. Virtualization software runs on the system hardware 10 and supports at least one Virtual Machine (VM) 18. As different virtualization functionalities are being implemented in hardware, including in recent microprocessor architectures and in recent I/O devices, the virtualization software may be referred to more broadly as virtualization logic 20. Thus, virtualization logic may comprise a wide variety of virtualization functionalities, whether the various functionalities are implemented in hardware, software or firmware.
Virtualization logic 20 comprises a VMKernel 22 and a Virtual Machine Monitor (VMM) 24. The VMKernel further comprises a disk device driver 46 appropriate for the storage adapter 16. The VMM further comprises one or more modules that emulate one or more virtual disks 28 and a virtual storage adapter 30 for use in or by the VM. The disk emulation functionality for emulating the virtual disk 28 may actually be implemented partially in the VMKernel and partially in the VMM. The VM 18 comprises virtual system hardware 32, including one or more virtual Central Processing Units (vCPUs) or virtual processors 34, virtual memory 36 and virtual storage adapter 30. A guest Operating System (OS) 38 runs on the virtual system hardware 32, along with one or more guest applications 40. The guest OS includes a disk device driver 26 appropriate for the virtual storage adapter 30. Although the virtual disk 28 is shown separate from the physical disk 12, the virtual disk may actually be implemented using portions of the physical disk.
In FIG. 1, if one of the guest applications 40 requires data from a file stored on the virtual disk 28, the guest OS 38 and device driver 26 will process the “data read”, and this virtual disk read will be conveyed to the virtual storage adapter 30. The VMM 24 and the VMKernel 22, including the disk/adapter emulator 44 and the device driver 46, convert the virtual disk data read to a corresponding disk read from the physical disk 12. This physical disk read is conveyed to the storage adapter 16. The data resulting from the physical disk read is written to physical memory 14 in a conventional manner and a physical disk I/O completion interrupt is raised at a physical CPU 48 in the system hardware 10. The VMKernel 22 determines the VM 18 to which the physical I/O completion relates and the VMKernel and the VMM 24 emulate a virtual disk I/O completion. This virtual disk I/O completion virtualizes a data read from the virtual disk 28 to the virtual memory 36. As a result, a virtual disk I/O completion interrupt is delivered to the virtual CPU 34. In response to the interrupt, the guest OS 38, including the device driver 26, provides the data to the application 40.
A virtualized computer system may be set up to provide high I/O rates. For example, the disk 28 may actually be a Storage Area Network (SAN) and the storage adapter 30 may actually be one or more Host Bus Adapters (HBAs). Many important datacenter applications today exhibit high I/O rates. For example, transaction processing loads can issue hundreds of very small I/O operations in parallel resulting in tens of thousands of I/Os per second (IOPS). Such high IOPS are now within reach of even more IT organizations with faster storage controllers, increasing deployments of high performance consolidated storage devices using SAN or Network-Attached Storage (NAS) hardware and wider adoption of solid-state disks.
In both virtualized and non-virtualized (physical) environments, at high I/O rates the vCPU or CPU overhead for handling all the interrupts may be high and can eventually lead to lack of CPU resources for the application itself. CPU overhead is even more of a problem in virtualization scenarios in which a goal is to consolidate as many virtual machines into one physical box as possible. Traditionally, interrupt coalescing or moderation has been used in storage controller cards to limit the number of times application execution is interrupted by a device to handle I/O completions. For interrupt coalescing, attempts are made to carefully balance the increase in I/O latency with the improved execution efficiency resulting from delivering fewer interrupts.
In hardware controllers, fine-grained timers may be used in conjunction with interrupt coalescing to establish an upper bound on the increased latency of coalescing I/O completion notifications. That is, a timer may be employed to fire a I/O completion interrupt if a time limitation has been reached since a last I/O completion interrupt. Such timers are difficult to implement and are inefficient to use in virtualization logic. This problem is challenging for other reasons, both in virtualized and physical environments.